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On-line quizzes are used to help first year University Mathematics students identify 
weaknesses in their basic skills and improve them. Quizzes developed as a formative tool 
have been utilised at JCU for eight years. However, before this research no-one has 
questioned the effectiveness of quizzes for this task. We present a description of the quizzes 
used in a core first year Mathematics subject at JCU and provide a statistical overview of 
their usage and efficacy for the intended task. 

Quizzes are a ubiquitous part of the assessment regime in tertiary education centres 
throughout the world. The advent of the web and Information Learning Technology (ILT) 
has transformed the traditional paper based quiz into a means of electronically assessing 
students and providing formative feedback. Several studies have shown positive correlations 
between student motivation, engagement and attitude and the online quiz environment, both 
in numerate (Angus and Watson, 2009) and literate (Johnson and Kiviniemi, 2009) 
disciplines. However, in the Mathematical disciplines there appears to have been little 
research that studies the direct relationship between the quiz questions and the student’s 
ability to reproduce the specific techniques that the quiz questions are based on at a later 
date. There also appears to be little available data on the way that students use this type of 
quiz. 

The online quizzes at James Cook University (JCU) were set up to provide a formative 
learning environment where first year Mathematics students could practise basic skills in 
their own time and pace, and allow students to develop mastery of basic building blocks in a 
specific subject. Online quizzes were provided in two traditional core Algebra and Calculus 
courses, as well as a Discrete Mathematics subject for IT students. These subjects have run 
for eight years, and student surveys reinforce the positive feedback reported in the literature. 
In particular, students believe that on-line quizzes increase their basic skill set and are a 
motivating factor in their Mathematical studies. However, the MathsART have raised 
several questions about the quiz environment. For example, ‘Do students help each other?’, 
‘Do students guess multiple choice questions?’, ‘Are we encouraging rote learning rather 
than understanding?’, ‘Are students mastering the basic skills required?’. 

The MathsART tried to answer some of these questions, and our initial investigations 
revealed how little we actually knew about the learning environment that students used with 
the online quizzes, so we decided to take a step back and collect basic data on the quiz 
environment at JCU. In this paper, we provide a statistical summary of the student usage of 
the on-line quizzes in one of the core Mathematics subjects. We present a basic statistical 
analysis of student’s performance on specific questions, presented both in the online quiz 
environment and in an invigilated environment. Students were asked to sign consent forms 
at the start of the subject, and 150 out of a cohort of 201 consented to the use of their data. 
One student did not complete any assessment leaving 149 students in the study. All the 
analyses presented in this paper are based on the results obtained for the consenting students 
that completed assessment. 

The rest of this paper is organized as follows. In the next Section, we describe the 
structure of the subject and the nature of the quiz environment. Next, we provide a basic 
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statistical analysis of the results for the quizzes and finally, discussion of these results and 
recommendations for future study. 

Quizzes and MAI 000 at JCU 

The online quizzes that are the focus of this paper form part of the learning and 
assessment environment for the first year, core Mathematics subject MAI 000: Mathematical 
Foundations. MA1000 is a one semester subject on Algebra, Analytic Geometry, 
Trigonometry and Introductory Calculus, and is worth 12.5% of the points required for the 
first year of a Science or Engineering degree. The assessment consists of nine on-line 
quizzes, three on-course tests and an end of semester exam. A student can pass the subject 
and avoid the end of semester exam, if they pass the on-course tests. Consequently, most 
students take the on-course assessment very seriously. Although the quizzes form part of the 
summative assessment for the subject, they are only worth 10%, which is given for 
encouragement and does not have any significant effect on the marks for the subject. 

Students are expected to access the online quizzes via LeamJCU, the Blackboard 
interface that JCU uses for online learning and reporting. Quizzes can be attempted from the 
JCU campus or remotely, and are not invigilated. The ten questions that comprise each of 
the nine quizzes are drawn randomly from a database of quiz questions. Students are 
encouraged to attempt a quiz as many times as they wish, and the highest score (out of ten) 
is recorded. 

A quiz is generated automatically when a student accesses the system, by randomly 
drawing ten questions from the database. The questions are of two basic types: multiple 
choice and numeric answer. For the multiple choice questions, a student is given a choice of 
four answers. For example, a multiple choice quiz question is 

Simplify: 

Which of the following expresses the answer with positive indices? 

(a) 2 3 (b) 2 2 (c) 2 n + 2 (d) n - 4 

The numeric questions were generated using “template” questions; a new quiz question 
is obtained by simply changing the numeric constants or the variable names in the template 
question. For example, a numeric quiz question is 

Determine the value of x that satisfies 

2 X+1 

— 1 

44 — 3x 

Each of these questions had sixteen variations, so each ten question quiz was randomly 
drawn from 160 possible choices. The questions on the quizzes are drawn from two 
categories, “simple drill” (SD) and “complex drill” (CD). Simple drill questions correspond 
to material that the student should be able to answer correctly 95% of the time, once they 
have completed that component of the subject, and these types of questions will be the main 
focus of this paper. 

Figure 1 is a plot of the average number of attempts that students make on each of the 
quizzes, assuming the student attempted the quiz. We note that fifty percent of the class 
attempted the quiz at least twice, with the exception of quiz six, which was attempted at 
least once. The maximum number of attempts was thirty, for quiz eight. Fifty percent of the 
class attempted quiz four at least three times. 

Students are expected to receive close to full marks for the quiz component of the 
course, and are expected to use the quizzes to master the material covered. Figure 2 is a plot 
of the average marks obtained for the class for each quiz, with 95% confidence intervals 
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marked. We note that the average mark is between 85% and 95%, except for quiz four, 
which was 80%. Quiz four was on the Binomial Theorem and the Binomial Series, which 
students found difficult. 




Figure 1. Number of attempts for each quiz. 


Figure 2. Average % obtained for each quiz. 


The Quiz Data and Analysis 

The data collected for the analyses presented in this paper is summarised as follows. 
Nine SD questions were selected and then examined in the oncourse tests. Four of these 
questions were also included in the end of semester examination. Each quiz question was 
worth one mark in the online quizzes. However, in the tests and examination the number of 
marks varied between two and three, so the averages presented are given in two ways: first, 
with the test and examination marks normalised so that they are out of one and secondly, 
with their original weighting. As noted in the previous Section, most students get close to 
full marks for each quiz, so we have calculated the average score for each question, where 
the average is taken over all of the student’s attempts. 

Table 1 presents the overall average marks (i.e., including SD questions) obtained in 
quizzes and the invigilated assessment for the subject. The subject is divided into three 
sections, each concluding with an on-course test. The quizzes for each section are completed 
before the test. The examination is at the end of the semester, for those students who do not 
elect to accept their grade on the oncourse assessment. Of the 149 students in our study, 37 
did not sit the final exam. 

Table 1 

Overall Average Percentages for the Questions in the Quizzes and the Tests/Examination 


Quiz Average 

Normalised Test/Exam Average 

Actual Test/Exam Average 

71.6% 

71.4% 

67.6% 


We note here that the quiz average and the normalised Test/Exam average are almost 
identical, with the actual Test/Exam average slightly smaller. Note that these results are 
significantly less than the target value of 95% and the average results for each quiz. 


Next, we examine the average marks obtained for each of the nine quiz questions used in 
this study. Figure 3 is a plot of the normalised Test/Exam average and the quiz average for 
each question. We haven’t included the actual Test/Exam averages, as they follow the same 
trend noted for the overall results: that is, approximately 5% less than the normalised results. 
Confidence intervals were not included, as the 95% intervals were almost too small to be 
seen on the plot. 
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Figure 3. Average marks for the quiz questions on the quizzes, tests and exams 

Close examination of Figure 3 reveals a different picture to the aggregated results 
presented in Table 1. First, the average scores for the quizzes are significantly higher than 
the overall averages for the tests and exams for most of the quiz questions. The lower 
averages for the overall results are probably due to the inclusion of the results for CD 
questions. The exceptions are the exam questions that were taken from quizzes completed 
earlier in the semester, and quizzes four, seven and eight. The average score for the quiz 
questions are a good predictor of the results for the tests for almost all the quiz questions. 
The only real exception is question eight, which significantly underestimates the test result. 

Discussion 

The analyses presented in this paper suggest that the overall quiz statistics (such as the 
quiz average) can be misleading. It would be easy to conclude from Figure 2 that the goal of 
95% correct for the SD questions was within reach, whereas the individual quiz question 
data reveals that it is not. Also, the performance on the examination for quiz questions that 
were taught early in the semester (questions two and four in Figure 3) suggest that students 
are forgetting material that they have dealt with successfully earlier in the subject. The 
results for individual questions also suggest that the distinction between SD and CD needs 
to be revisited, as the average marks for questions four, seven and eight imply that these are 
not SD questions. These results are only preliminary, and more research needs to be carried 
out on the relationship between the average score per question and performance on test and 
exam results. 
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